Data mining systems and methods for heterogeneous data sources

ABSTRACT

Systems and methods integrate disparate backup devices with a unified interface. In certain examples, a management console manages data from various backup devices, while retaining such data in its native format. The management console can display a hierarchical view of the client devices and/or their data and can further provide utilities for processing the various data formats. A data structure including fields for storing both metadata common to the client device data and value-added metadata can be used to mine or process the data of the disparate client devices. The unified single platform and interface reduces the need for multiple data management products and/or customized data utilities for each individual client device and provides a single pane of glass view into data management operations. Integrating the various types of storage formats and media allows a user to retain existing storage infrastructures and further facilitates scaling to meet long-term management needs.

CROSS-REFERENCE TO RELATED APPLICATION

Any and all applications for which a foreign or domestic priority claimis identified in the Application Data Sheet as filed with the presentapplication are hereby incorporated by reference under 37 CFR 1.57.

This application is also related to the following U.S. patentapplications filed on Mar. 30, 2010, each of which is herebyincorporated herein by reference in its entirety:

-   -   U.S. patent application Ser. No. 12/749,862, now published as        U.S. Patent Application Publication No. 2010/0332990, entitled        “Information Management Systems and Methods for Heterogeneous        Data”; and    -   U.S. patent application Ser. No. 12/749,897, now Issued as U.S.        Pat. No. 8,352,432, entitled “Systems and Methods for        Normalizing Data of Heterogeneous Data Sources”.

BACKGROUND

Field

Embodiments of the invention relate to information management and, inparticular, to systems and methods for managing data from heterogeneousdata sources.

Description of the Related Art

As the reliance on electronic information increases, individuals andbusinesses are turning more and more to various backup data solutions.Conventional backup systems generally employ a monolithic backup andretrieval system servicing a single server with attached storagedevices. These systems usually control all aspects of a data backup orretrieval and often direct only one type of backup, whether it is anetwork backup or a single machine backup.

With the numerous available backup solutions also comes the challengeassociated with accessing and utilizing the stored data, especially whendata is stored in different backup formats in a distributed computingenvironment. For instance, in a business enterprise, different divisionswithin the same company may use different vendors and/or applications toback up, or otherwise store, data over a network having multipleinterconnected computers, often heterogeneous and/or geographicallyremote.

Certain information management systems attempt to address at least someof the foregoing problems by using a common application to back up datafrom a variety of sources (e.g., SQL server, MICROSOFT EXCHANGE) suchthat the all the data in the enterprise environment is stored in thesame backup format. For example, the GALAXY system commerciallyavailable through CommVault Systems, Inc. (Oceanport, N.J.) supportsbackup of data through a common application that allows for subsequentdata retrieval, reporting and information processing. However, suchsystems still require data from the various sources to be converted froma native format to a different single backup format.

SUMMARY

In view of the foregoing, a need exists for information managementsystems and methods that ingest data in a variety of formats, includingdifferent backup formats, for access and/or management through a singleview. For example, in certain embodiments of the invention, a system isdisclosed that obtains data from heterogeneous backup sources and thatallows a user (e.g., a system administrator) to access, catalog and/ormanipulate the data through a single interface while, at the same time,allowing the backup sources to retain the data in their native backupformats.

Certain systems and methods are disclosed herein for integrating aplurality of disparate backup devices with a unified graphical userinterface. In certain embodiments, a management console manages datafrom various backup devices, while retaining such data in its nativebackup format. For example, the management console can comprise a windowthat displays a hierarchical view of manageable client devices on anetwork and/or data associated therewith. Moreover, the console canfurther provide options through which the console user may navigate toprocess the data of one or more different devices. For instance, theuser may be able to “drill down” through the hierarchy of clientdevices. In certain embodiments, the user interface can display varioustypes of reporting information, such as graphs, alerts, charts,diagrams, textual reports or the like, based on one or more rules orpolicies applied to the data of the client devices.

For example, certain embodiments of the invention provide a unifiedsingle platform and interface that reduces the need for multiple datamanagement products and/or customized data utilities for each individualclient device and that provides a single pane of glass view into datamanagement operations. Such embodiments can advantageously integratevarious types of storage formats and media, thereby allowing a user toretain existing storage infrastructures and further facilitating scalingto meet long-term management needs.

In certain embodiments, system and methods disclosed herein can create aunified policy management framework for multiple pre-certified storagevendors or storage formats and offer access to data on various storagetiers (e.g., online, nearline, offline). Certain embodiments furtherinclude offline mining tools for non-backup data, such as EXCHANGE,SHAREPOINT or ACTIVE DIRECTORY data, to recover granular messages anddocuments directly from database recovery copies.

In certain embodiments, a method is disclosed for administering astorage network comprising heterogeneous client devices. The methodincludes displaying with a management console information regardingbackup data stored in a plurality of heterogeneous client devices,wherein the backup data of each client device is stored in a differentbackup format, and wherein the management console integrates a pluralityof utilities for performing a plurality of data management operations onthe backup data of the heterogeneous client devices, wherein themanagement console further provides a single interface for the pluralityof utilities. In further embodiments, the method also includes selectinga first data management operation of the plurality of data managementoperations to be performed by at least one of the plurality of utilitiesin response to user input and selecting backup data from at least one ofthe plurality of heterogeneous client devices in response to user input.Finally, the method can include performing the first data managementoperation on the selected backup data of the at least one client devicein response to said selecting the first data management operation andsaid selecting the backup data.

In certain embodiments, a system is disclosed for administering astorage network comprising heterogeneous client devices. The systemincludes a plurality of data connectors and a management console. Thedata connectors are in communication with a plurality of heterogeneousclient devices, each of the plurality of data connectors beingconfigured to obtain data from a respective one of the plurality ofheterogeneous client devices. The management console executes on acomputing device in network communication with the plurality of dataconnectors. The management console is further configured to: provide aplurality of utilities for performing data management operations on thedata obtained by the data connectors; display via a unified interfaceon-screen graphics indicative of each of the heterogeneous clientdevices and the utilities; receive a selection of a first datamanagement operation to be performed by at least one of the utilities onthe data of at least one of the heterogeneous client devices; andperform the first data management operation on the selected data of theat least one heterogeneous client device.

In certain further embodiments, a system is disclosed for administeringa storage network comprising heterogeneous client devices. The systemincludes means for obtaining data from a plurality of heterogeneousclient devices storing data in a plurality of different data formats andmeans for providing a plurality of utilities for performing a pluralityof data management operations on the data obtained by said obtainingmeans. The system further includes means for displaying via a unifiedinterface on-screen graphics indicative of each of the heterogeneousclient devices and the utilities and means for receiving a selection ofa first data management operation to be performed by at least one of theutilities on the data of at least one of the heterogeneous clientdevices. The system further includes means for performing the first datamanagement operation on the selected data of the at least oneheterogeneous client device.

In addition, a method is disclosed for managing data in a computersystem. The method includes: obtaining first data in a first file formatfrom first media, the first data being associated with a first payloadand first metadata; processing the first payload to generate secondmetadata different than the first metadata; and storing in a datastructure the second metadata and at least a portion of the firstmetadata in a first entry associated with the first media. The methodfurther includes: obtaining second data in a second file format fromsecond media, the second data being associated with a second payload andthird metadata, the second file format being different than the firstfile format; processing the second payload to generate fourth metadatadifferent than the third metadata; and storing in the data structure thefourth metadata and at least a portion of the third metadata in a secondentry associated with the second media. The method can further includepresenting the first and second entries in a unified view of a userinterface, such as in a single browser window.

In certain embodiments, a system is disclosed for managing data from aplurality of heterogeneous client devices. The system comprises firstand second data connectors, a management module and a data structure.The first data connector obtains first data in a first file format froma first client device, the first data having a first payload and firstmetadata associated with the first payload. The second data connectorobtains second data in a second file format from a second client device,the second data having a second payload and second metadata associatedwith the second payload, the second file format being different than thefirst file format. The management module executes on a computing devicein communication with the first and second data connectors to receivethe first and second payloads and first and second metadata. Themanagement module can further process the first payload to generatethird metadata different than the first metadata and to process thesecond payload to generate fourth metadata different than said secondmetadata. A data structure, such as an Extensible Markup Language (XML)file, is configured to store the third metadata and at least a portionof the first metadata in a first entry associated with the first clientdevice and to store the fourth metadata and at least a portion of thesecond metadata in a second entry associated with the second clientdevice.

In certain embodiments, a system is disclosed for managing data from aplurality of heterogeneous client devices. The system includes firstmeans for obtaining first data in a first file format from a firstclient device, the first data having a first payload and first metadataassociated with the first payload. The system also includes second meansfor obtaining second data in a second file format from a second clientdevice, the second data having a second payload and second metadataassociated with the second payload, the second file format beingdifferent than the first file format. The system further includes meansfor receiving the first and second payloads and first and secondmetadata and for processing the first payload to generate third metadatadifferent than the first metadata and to process the second payload togenerate fourth metadata different than the second metadata. Inaddition, the system can include means for storing the third metadataand at least a portion of the first metadata in a first entry associatedwith the first client device and to store the fourth metadata and atleast a portion of the second metadata in a second entry associated withthe second client device.

In certain embodiments, a method is disclosed for processing data from aplurality of heterogeneous client devices. The method comprisesproviding a single interface including on-screen graphics representing aplurality of heterogeneous client devices in a storage network system,wherein the heterogeneous client devices comprise first client devicesstoring backup data in different backup formats and second clientdevices storing non-backup data. The method further includes receiving adata operation request including (i) a selection of at least one secondclient device, and (ii) a first data management operation to beperformed on the non-backup data of the at least one second clientdevice. The method also includes copying the non-backup data of the atleast one second client device to a first memory location; emulating ona proxy device a native application associated with the non-backup dataof the at least one second client device; and processing with theemulated native application the first data management operation on thecopied non-backup data and storing the processed, copied non-backup dataon a storage device.

In certain further embodiments, a system is disclosed for administeringa storage network comprising heterogeneous client devices. The systemcan include first and second client devices, a proxy device, a secondarystorage device and a management console. The first client devices areconfigured to store backup data in different file formats, and thesecond client devices are configured to store non-backup data indifferent file formats. The proxy device includes at least one emulatednative application executing thereon, and the management console isconfigured provide a single interface including on-screen graphicsrepresenting the first and second client devices. The management consolecan be further configured to receive a first data operation request withrespect to the non-backup data stored on at least one of the secondclient devices and to copy the non-backup data of the at least onesecond client device to a staging location associated with the proxydevice. The proxy device can be further configured to process thenon-backup data of the at least one second client device with the atleast one emulated native application and to copy the processednon-backup data to the secondary storage device.

In certain embodiments, a system is disclosed for administering astorage network comprising heterogeneous client devices. The systemincludes first means for storing backup data in a plurality of firstfile formats and second means for storing non-backup data in a pluralityof different second file formats. The system also includes means forproviding a single interface including on-screen graphics representingsaid first and second storing means and for receiving a first dataoperation request with respect to the non-backup data stored on saidsecond storing means. The system further includes means for processing acopy of the non-backup data with at least one emulated nativeapplication and third means for storing the processed copy of thenon-backup data.

In certain embodiments, a method is disclosed for managing data in acomputer system. The method comprises obtaining first data in a firstfile format from first media, the first data being associated with afirst payload and first metadata. The method further comprisesprocessing the first payload to generate second metadata different thanthe first metadata and storing in a database the second metadata and atleast a portion of the first metadata in a first entry associated withthe first media. The method further comprises obtaining second data in asecond file format from second media, the second data being associatedwith a second payload and third metadata, the second file format beingdifferent than the first file format. The method also comprisesprocessing the second payload to generate fourth metadata different thanthe third metadata and storing in the database the fourth metadata andat least a portion of the third metadata in a second entry associatedwith the second media. Finally, the method can further present the firstand second entries in a unified view (e.g., single browser window) of auser interface.

For purposes of summarizing the disclosure, certain aspects, advantagesand novel features of the inventions have been described herein. It isto be understood that not necessarily all such advantages may beachieved in accordance with any particular embodiment of the invention.Thus, the invention may be embodied or carried out in a manner thatachieves or optimizes one advantage or group of advantages as taughtherein without necessarily achieving other advantages as may be taughtor suggested herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary block diagram of an informationmanagement system, according to certain embodiments of the invention.

FIGS. 2A through 2D illustrate exemplary embodiments of simplified datastructures usable for storing data and/or metadata extracted from one ormore client devices of the information management system of FIG. 1.

FIG. 3 illustrates a block diagram of an exemplary embodiment of theinformation management console of the information management system ofFIG. 1.

FIG. 4A illustrates an exemplary simplified menu usable with the userinterface of the information management system of FIG. 1.

FIGS. 4B through 4F illustrate exemplary screen displays usable with theuser interface of the information management system of FIG. 1.

FIG. 5 illustrates a flowchart of an exemplary process for managing datawith the information management system of FIG. 1.

FIG. 6 illustrates a flowchart of an exemplary process for processingdata requests with the information management system of FIG. 1.

FIG. 7 illustrates an exemplary block diagram of an informationmanagement system for merging disparate data of multiple client devices,according to certain embodiments of the invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Embodiments of the invention disclosed herein are advantageously used tomanage data from heterogeneous sources and formats in a computer networkenvironment. In certain embodiments, an information management consoleor appliance communicates with multiple client devices, such asdifferent backup servers and/or databases, to present heterogeneous datato a user in a manageable format through a unified view. Thus, data inthe information management system can be stored in different backupformats (e.g., by different backup applications) and/or non-backupformats, while also being capable of being displayed to the user in asingle view for further data processing and/or manipulation (e.g.,restoring, archiving, searching or the like). As a result, data instored on the client devices is not required to be converted to adifferent format through a common backup or archive application prior tobeing managed by the information management console.

In certain embodiments of the invention, the information managementconsole can utilize individual agents or data connectors that arefamiliar with the data characteristics and formats of different filetypes to obtain file data, metadata and/or configuration settings fromthe client devices. Moreover, the user is able to manage all data typesin the information management system through a single interface. Incertain embodiments, metadata associated with the data on theheterogeneous client devices is normalized for presentation to the usersuch as through one or more reports.

In certain embodiments, the versatility of the disclosed informationmanagement systems and methods allows for flexibility in managing datain an enterprise environment. For instance, disclosed systems andmethods allow for the dynamic addition or removal of client devices fromthe system, application of particular retention policies or otherstorage polices across select data from a plurality of heterogeneousdata sources, access to data of one or more databases without requiringthe native database application to be online, restoration of data whosebackup application has been decommissioned, combinations of the same orthe like.

The features of the systems and methods will now be described withreference to the drawings summarized above. Throughout the drawings,reference numbers are re-used to indicate correspondence betweenreferenced elements. The drawings, associated descriptions, and specificimplementation are provided to illustrate embodiments of the inventionand not to limit the scope of the disclosure.

In addition, methods and functions described herein are not limited toany particular sequence, and the blocks or states relating thereto canbe performed in other sequences that are appropriate. For example,described blocks or states may be performed in an order other than thatspecifically disclosed, or multiple blocks or states may be combined ina single block or state.

FIG. 1 illustrates a block diagram of an information management system100, according to certain embodiments of the invention. In certainembodiments, the information management system 100 provides a commoninterface for the management of data stored on heterogeneous datasources and in a variety of file formats, such as data in differentbackup and/or non-backup formats. For example, as described in furtherdetail below, certain embodiments of the information management system100 normalize disparate backup data and accompanying metadata from avariety of data sources. This normalized data is then advantageouslycombined with additional metadata to be displayed and/or managed via acommon, unified graphical user interface (GUI).

With reference to FIG. 1, the information management system 100 includesan information management appliance or console 102 that communicateswith a plurality of data sources or client devices 104 a, 104 b, 104 c,. . . 104 n (hereinafter “104”). In certain embodiments, the informationmanagement console 102 comprises one or more data agnostic servercomputers that can receive ingested data and/or perform various datamanagement functions or utilities disclosed herein. For instance, theinformation management console 102 can comprise a plurality of modulesexecuting on the one or more server computers to ingest heterogeneousdata from a variety of backup and/or database sources and present thedata to the user via a common format. In yet other embodiments, theinformation management console 102 can comprise one or moreworkstations, personal computers, laptops, notebooks, processors,computing devices, virtual machines, combinations of the same or thelike.

The information management console 102 communicates with each of theplurality of client devices 104 to obtain the data to be managed. Incertain embodiments, the client devices 104 form part of a businessenterprise system and/or comprise heterogeneous sources of data. Forinstance, the client devices 104 can comprise servers from differentbackup vendors (e.g., COMMVAULT, VERITAS, NETAPP, EMC, COMPUTERASSOCIATES) that each back up data in a different format to one or moremedia devices 112 a, 112 b, 112 c, . . . 112 n (hereinafter “112”)(e.g., secondary storage devices). In such embodiments, the informationmanagement console 102 can communicate directly with the clientdevice(s) 104, directly with the media device(s) 112, and/or with themedia device(s) 112 through the client device(s) 104, a proxy serverand/or other network path.

In certain embodiments, one or more of the client devices 104 cancomprise a database server or other collection of non-backup informationaccessible by the information management console 102. For instance, thedatabases can be associated with a variety of database applications,such as, for example, EXCHANGE SERVER, SHAREPOINT SERVER, SQL SERVER,LOTUS NOTES, ORACLE applications, SAP applications or the like. Incertain embodiments, the information management console 102 can accessthe databases via a database management system (DBMS).

It will be appreciated that the client devices 104 can comprise datastored on a variety of physical media and/or virtual devices, such asmagnetic disk, tape, cloud storage, filers, combinations of the same orthe like. In certain embodiments, the client devices 104 can comprisevirtual machines executing on one or more host servers.

Although media devices 112 are illustrated separate from the clientdevices 104 in FIG. 1, it should be appreciated that the media device112 can also be integrated with client device 104, and/or multipleclient devices 102 can share a common media device 112. Thus, the term“client device” as used herein is a broad term and is used in itsordinary sense and includes, without limitation, any source of data,database or data repository, or a computing device (e.g., as astand-alone complete entity and/or distributed over multiple entities)that can access such data.

For example, in embodiments in which the client device 104 comprises abackup server or the like, it is appreciated that the client device 104can coordinate data retrieval from a plurality of storage or mediadevices to be transferred to and/or managed by the informationmanagement console 102 through the client device 104. Thus, theinformation management console 102 need not read directly from, orcommunicate with, all storage devices 112 that contain data managed bythe information management console 102. Moreover, a single client devicecan comprise a server or other computing device that communicates with aplurality of additional client devices (e.g., sub-clients) coupledthereto.

Each of the client devices 104 is further associated with at least oneof the agent or data connectors 106 a, 106 b, 106 c, . . . 106 n(hereinafter “106”) for gathering and providing relevant informationfrom the client device 104 to the information management console 102. Incertain embodiments, each data connector 106 is configured to perform adiscovery of the data on the client device 104, such as data produced byone or more backup applications (e.g., SIMPANA (CommVault Systems,Inc.), NETBACKUP (Symantec Corporation), NETWORKER (EMC Corporation)).In general, during this discovery process, the data connector 106retrieves data and/or metadata from the client device 104 and molds theretrieved information into a predetermined and/or normalized data format(e.g., the GALAXY data format provided by CommVault Systems, Inc.).

In certain embodiments, each connector 106 is specific to a particulartype of data format and/or client device 104. For instance, each clientdevice 104 can have a plurality of application-specific connectors 106associated therewith for handling different data types stored on thedevice 104. For example, one connector 106 can be configured to handlebackup data, while another connector 106 is configured to track systemconfigurations of the client device 104.

In certain embodiments, when the client device 104 is initiallyinstalled in, or connected to, the information management system 100, auser can be prompted, such as through a user interface 108, forinformation (e.g., configuration information) regarding the dataformat(s) used by the client device 104. For example, when a clientdevice 104 comprising an EXCHANGE SERVER is initially installed in theinformation management system 100, the user can input informationregarding user mailboxes, snapshot information, security and/orauthentication information, combinations of the same or the like.Following the entry of the configuration information, the appropriatedata connector 106 can be push-installed on the client device 104 fromthe information management console 102. In yet other embodiments, thedata connector 106 can be located external to the client device 104,installed on one or more media devices 112 of the client device 104,and/or communicate remotely with the client device 104 via remoteprocedure calls (RPCs), an application programming interface (API), acomponent object model (COM) interface, combinations of the same or thelike.

In yet other embodiments, the data connector 104 can be configured toscrape transaction logs, execute database commands, issue queries,monitor filter drivers and/or registry information, read backup files,or the like, to obtain pertinent information from the client device 104.In yet other embodiments, the data connector 104 can function as one ofthe clients, read third-party metadata, read XML files, parse INI orother configuration files, combinations of the same or the like, toobtain information about data on the client device 104.

In certain embodiments, the connectors 106 provide file data to theinformation management console 102. For example, the connectors 106 canprovide copies of select data from the client devices 104 based on oneor more requests from the information management console 102. In certainembodiments, the connectors 106 are configured to provide metadata tothe information management console 102 in addition to, or in place of,actual file data. For example, the connectors 106 can provideinformation regarding: identification of media (e.g., tapes, magneticdisks) containing the file data, identification of client devices ormachines 102 associated with the data, creation date, modification date,file type, file size, owner, permissions, native data retention polices,storage locations, jobs, schedules, existing and/or scheduled snapshots,protection of databases, mailbox information, combinations of the sameor the like. In yet further embodiments, the connectors 106 can providestatistics regarding use of a particular client device 104 or groups ofdevices, such as, for example, daily data growth on machine, number ofusers, new mailboxes setup or the like.

In certain preferred embodiments, the data connectors 106 obtain the rawdata and/or metadata in the native format of the application(s) used bythe particular client device 104 and can push the information to theinformation management console 102. This advantageously allows for usersto retain backup data from a plurality of vendors without requiring allthe data to be copied or transformed into a different format. Theobtained information is advantageously input into a normalized datamodel or structure 109 that includes multiple fields that are common toa variety of data formats. In certain embodiments, the data structure109 advantageously provides a common schema or platform with which tomanage and/or manipulate data from a variety of different data formatsand/or heterogeneous or homogeneous client devices 104.

For instance, in certain embodiments, the normalized data structure 109can comprise an XML file that receives ingested data from a plurality ofdata connectors 106. Simplified examples of such XML files are shown anddescribed with reference to FIGS. 2A-2D. In yet other embodiments, eachclient device 102 and/or data format is associated with a single datastructure 109.

As further illustrated in FIG. 1, the normalized data structure 109 ismaintained by the information management console 102. In certainembodiments, the data structure 109 can be stored on the informationmanagement console 102, a management database 110 and/or a secondarydatabase 111. In yet other embodiments, the data structure 109 can bemaintained and/or stored by the individual data connectors 106, on acentral storage device, or the like. In certain embodiments, the datastructure 109 can comprise an index, a table, combinations of the sameor the like.

In certain embodiments, the connectors 106 are capable of querying oneor more client devices 104 and/or media devices 112 for higher-levelmetadata without needing to enter or touch the actual media. Forexample, the connectors 106 can interface with one or more APIsassociated with the particular software platform executing on the clientdevice 104. In yet other embodiments, the connectors 106 read directlyfrom the media of the client device 104 to obtain the necessary dataand/or metadata.

In certain embodiments, the connectors 106 are configured topersistently and/or periodically monitor the client devices 104. Suchmonitoring can include, for example, tracking information relating toone or more jobs and/or media associated with a particular client 104.In yet other embodiments, the connectors 106 can be configured toexecute on demand, such as in response to instructions from a user,specified policy or rule.

In certain embodiments, the client devices 104 advantageously retain thedata stored thereon in the native backup format(s). That is, the dataconnector 106 or information management console 102 does notsubstantially modify or entirely copy the data stored on the clientdevice 104. In yet further embodiments, “foreign” or “native”applications running on the client device 104 are monitored by theconnector 116 to identify changes to the data. Moreover, in suchembodiments, if backup data from the client device 104 is to be restoredfor further processing, the foreign application can be used to restorethe data, such as to a temporary staging area.

The user interface 108 advantageously provides a unified view of thedata within the information management system 100 so as to allow a user(e.g., a system administrator) to manage and/or view data from multipleclients 104. For example, the user interface 108 can provide the userwith a single view and access to the data of heterogeneous sourcedevices, such as devices with backup data from different backup vendors.

The user interface 108 can also allow the user to manage data that hasbeen cataloged on a plurality of different source devices 104, even ifthe data is stored in different backup or other file formats. In certainembodiments, the user interface is populated with data acquired from, orbased on information stored in, the normalized data structure 109. Forinstance, in certain embodiments, the user interface 108 can provide theuser with the option of creating copies of select portions of data fromdifferent source devices 104 that have been cataloged or scanned by thedata connector 106.

The ability to process and retrieve select pieces of data from multipleclient devices 104, each with data in its own native format, providessignificant advantages to the information management system 100. Forexample, a storage administrator wanting to back up files, documentsand/or email that contain a particular set of words (or other criteria),can submit his or her data request via the user interface 108.

In certain embodiments, the user can select which of the differentclient devices 104 are to be associated with the data request. Theinformation management console 102 can then instruct the differentconnectors 106 associated with the select client devices 104 to obtainthe requested data. Because each connector 106 is familiar with the fileformat of the data on the corresponding client device 104, theconnectors 106 are able to obtain the requested data and transmit thedata from the different backup sources or file formats to theinformation management console 102 to be backed up. In yet otherembodiments, the connectors 106 can send copies of the entire data fromthe selected client device 104 to the information management console 102for further processing and selective backing up of data portions.

In certain embodiments, the information management console 102 stores ina management database 110 information regarding the client devices 104,jobs and/or media associated with one or more backup files and/ordatabases. For instance, the database 110 can maintain a map (e.g., atable or other data structure) of client devices 104 to particular jobsrunning in the information management system 100 based on informationgathered by the connectors 106.

In certain embodiments, the database 110 maintains a mapping of aplurality of heterogeneous backup environments (e.g., client devices 104from different vendors). This mapping, in certain instances, can bedisplayed through the user interface 108 to provide a user with a singleview of the backup jobs being performed and/or scheduled within theinformation management system 100. In such embodiments, a user canadvantageously determine if the backups from different vendors allcomply with applicable storage polices and/or guidelines.

In yet other embodiments, the database 110 can store select portions offile data obtained by the connectors 106. For example, as discussedabove, a user may be given the option through the user interface 108 tospecify which types of data should be part of a backup and/or migrationoperation for long term storage (e.g., to satisfy certain legal holdrequirements, storage policies, user preferences or the like). Thesedata portions can be obtained by the information management console 102through the connectors 106 to be stored in the database 110. In yetother embodiments, the management database 110 can be configured tostore pointers or other links to the pertinent file data that is storedon and/or maintained by one or more of the client devices 104. Suchembodiments are discussed in more detail below with respect to FIG. 7.

In further embodiments, the database 110 can store a plurality of rulesand/or policies for managing the data of the various client devices 104.For instance, the database 110 can store information regarding thefrequency of scanning the client devices 104 with the connectors 106.Other rules or policies can provide established plans or conditionsregarding data retention, security, life cycle management, or the like.

In certain embodiments, at least a portion of the database 110 comprisesa metabase that stores metadata associated with files on the clientdevice(s) 104. Additional details regarding metabases usable withembodiments of the invention are disclosed in U.S. patent applicationSer. No. 11/563,940, filed Nov. 28, 2006, now published as U.S. PatentApplication Publication No. 2007-0179995 A1, which is herebyincorporated herein by reference in its entirety to be considered partof this specification.

As shown, the information management system 100 further comprisessecondary storage 111 in communication with the information managementconsole 102. In certain embodiments, the secondary storage 111 serves asa repository for data processed by the information management console102.

In yet other embodiments, the secondary storage 111 can serve as astaging area for data to be processed by the information managementconsole 102. For example, select client data needing to be temporarilyrestored or copied for particular data management or manipulationoperations can be moved to the secondary storage 111.

It should also be understood from the disclosure herein that themanagement database 110 and/or secondary storage 111 can each comprise aplurality of databases and/or data storage structures in one or morelocations, or can be combined into a single storage device or database.For example, a portion of the database 110 can be configured to storeactual file data (e.g., archive data), while another portion of thedatabase 110 can be configured to store metadata and/or configurationsettings.

One or more of the components of the information management system 100can advantageously communicate with each other through a network orother means for communicating. In certain embodiments, the networkcomprises a public network such as the Internet, virtual private network(VPN), token ring or TCP/IP based network, wide area network (WAN),local area network (LAN), an intranet network, point-to-point link, awireless network, cellular network, wireless data transmission system,two-way cable system, interactive kiosk network, satellite network,broadband network, baseband network, combinations of the same or thelike.

As discussed above, the information management system 100 is configuredto enrich the information extracted from the data on the heterogeneousclient device(s) 104. For instance, the information management console102 and/or data connector(s) 106 can supplement the normalized datastructure 109 with customized metadata related to the extractedinformation and/or client device 104. In certain embodiments, thevalue-added information can be common to various types of data formats.

FIGS. 2A-2D illustrate exemplary embodiments of simplified datastructures usable in the information management system 100 for storingdata and/or metadata extracted from heterogeneous client devices 104.For example, the illustrated data structures can correspond to thenormalized data structure(s) portrayed and described with respect toFIG. 1. In certain embodiments, the data structures advantageouslyprovide means for processing data of the heterogeneous client devices104 to be displayed on a common interface, manipulating such data withsingle queries that span the multiple client devices 104 and the like.

In particular, FIG. 2A illustrates an exemplary XML data structure 209 afor capturing information regarding data on one or more client devices104. In certain embodiments, the data structure 209 a is stored in adatabase accessible by the information management console 102 andcomprises multiple fields that provide for the normalization of datafrom multiple foreign data repositories, such as both backup andnon-backup (e.g., database server) data.

As shown in FIG. 2A, the data structure 209 a comprises informationregarding a client device 104 utilizing the SNAPMANAGER FOR MICROSOFTEXCHANGE storage management program offered by NetApp, Inc. Forinstance, the data structure 209 a includes entries regarding theoperating system being used by the client device 104, passwords,directory information, application information and the like.

FIG. 2B illustrates an exemplary XML data structure 209 b for capturinginformation regarding one or more jobs associated with the clientdevice(s) 104. For example, the data structure 209 b includes entriesregarding the types of storage operations to be performed on the clientdevice data, storage policy information, data locations and the like.

FIG. 2C illustrates an exemplary XML data structure 209 c for capturinginformation regarding one or more snapshots associated with data of theclient device(s) 104. In particular, the data structure 209 c containsinformation regarding snapshots taken with a NETAPP software program.For example, the data structure 209 c includes entries regarding thesnapshot creation time, snapshot location, source data location and thelike.

FIG. 2D illustrates an exemplary XML data structure 209 d for capturinginformation regarding one or more storage policies associated with theclient device(s) 104. In particular, the data structure 209 d includesentries regarding storage policies associated with the NETAPP storageplatform, including the client device 104 to which the storage policyapplies, descriptions of the data copies, storage retention information,and the like.

Although the data structures 209 a-209 d have been described withreference to particular embodiments, it should be understood that othertypes of data structures and/or data fields can be used to storeingested data and/or metadata obtained from the client device(s) 104.For example, the data structure 209 can include fields related toencryption information, compression, information lifecycle management,user permissions and like metadata.

FIG. 3 illustrates an exemplary block diagram of further details of aninformation management console 302, according to certain embodiments ofthe invention. In general, the information management console 302 isconfigured to ingest, catalog and manage data from a plurality ofheterogeneous sources, such as backup servers using different backupformats. For instance, in certain embodiments, the informationmanagement console 302 can be used in the information management system100 of FIG. 1 to provide a single pane view of the data of the pluralityof client devices 104.

For exemplary purposes, the management console 302 will be describedwith reference to a plurality of modules executing on one or morephysical computing devices that interact with each other to ingest,prepare and manipulate data with respect to the client devices 104. Itshould be understood that, in other embodiments, multiple modules can becombined into a single module, or one module can be separated intoadditional modules to perform substantially the same operations.Moreover, in other embodiments, one or more of the below-describedmodules can be implemented in a data connector 106, on a separate serveror the like.

As shown, the information management console 302 comprises an ingestionmodule 320 that receives the data obtained from client devices 104 bythe data connectors 106. As discussed above, in certain embodiments, theingested data is obtained from heterogeneous data sources and stored inone or more normalized data structures 109 according to a predefinedstorage policy, user preference and/or format.

In certain further embodiments, the ingestion module 320 is configuredto receive and/or obtain metadata regarding the files/folders on theclient devices 104. For example, the connectors 106 can obtain variouslevels of metadata depending on the preferences of the user and/ormakeup of the information management system 100. In certain embodiments,the ingestion module 320 first receives higher-level metadata regardingthe client system data and can subsequently instruct the data connectors106 to obtain additional lower-level metadata upon specific requests bythe user. Moreover, in certain embodiments, the ingestion module 320 isconfigured to scrub incoming data files for metadata (e.g., file type,user permissions, security settings), relevant data content,combinations of the same or the like.

A conversion module 322, in certain embodiments, receives and normalizesmetadata obtained by the ingestion module 320. For instance, theconversion module 322 can convert and/or associate ingested metadatawith a specific data model, such as predetermined data fields of thenormalized data structure 109. Moreover, in certain embodiments, theconversion module 322 can further enrich the information stored in thedata structure 109 based on the ingested data. For instance, theconversion module 322 can, based on the ingested data of a particularclient device 104, generate additional metadata (value-added metadata)that is not produced or maintained by the native applications of theclient device 104. The conversion module 322 can then store thisadditional metadata in the normalized data structure 109 and repeat theprocess for ingested data of different client devices 104. In thismanner, the normalized data structure 109 can include additionalmetadata that is common for all data types of the client devices 104.

A configuration module 324, in certain embodiments, maintains theconfiguration settings for each of the plurality of client devices 104.For instance, the configuration settings can inform the informationmanagement console 302 of backup schedules, retention policies, or thelike associated with backup devices in the information management system100. The configuration module 316 can also communicate with an interfacemodule 318 to provide a list of user utilities or data operationsavailable with respect to certain types of data. For instance, for aMICROSOFT EXCHANGE client device, the user may be provided withinformation regarding different storage groups within the EXCHANGEserver, users within the EXCHANGE server, or the like, based on settingsmaintained by the configuration module 324.

In certain embodiments, the configuration module 324 receives settingsinput by the user through the user interface 108. In yet otherembodiments, the configuration module 324 can receive such settings fromthe data connectors 106, the ingestion module 320, or from data storedin the management database 110.

The interface module 326 transmits data to and receives data from theuser interface 108. In certain embodiments, the interface module 326formats information received from the database 110 for graphicalpresentation to the user via an external display, such as through amenu, graph, report, window or the like.

The information management console 302 further comprises one or morerestore modules 328. In certain embodiments, the restore modules 328 areconfigured to restore data from one or more identified media deviceswithout the need for the native backup application to be online. Incertain embodiments, each restore module 328 is specific to a particulartype of data, such as a backup data format.

For instance, in certain embodiments, the user can select through theuser interface 108 to restore data of one or more files that are backedup in a format that corresponds to a backup application that has beendecommissioned. In such embodiments, the restore module 328, which canrecognize the backup format and appropriately process the data, canidentify and copy select portions of the backed up data. In certainembodiments, the restore module 328 can access the data directly afterthe media is mounted by the information management console 302, and/orthe restore module 328 can access the backed-up data through a proxyserver (e.g., by emulating a native application) or other pathway.

The information management console 302 further comprises one or moremining modules 330 that are configured to mine or process particulartypes of data. For instance, the mining module(s) 330, in certainembodiments, can break down a database into individual elements on agranular level (e.g., mailboxes, email messages, storage accounts, useraccounts, etc.) for additional cataloging, data processing, copying orthe like. In certain embodiments, each mining module 330 is specific toa particular type of data (e.g., EXCHANGE data, SHAREPOINT data, or thelike).

Although the information management console 302 has been described withreference to particular embodiments, other arrangements andconfigurations of the information management console 302 can be used.For instance, the information management console 302 can operate withfewer modules than those illustrated in FIG. 3. In yet otherembodiments, the information management console 302 can compriseadditional modules that perform additional data processing on clientdevice data, information stored in the normalized data structure(s) 109,staged data that has been restored from a client backup file, or thelike. Moreover, such data operations can be based on one or more userqueries or commands received from the interface module 326, storagepolices, rules or the like, and can include, for example: archiving,searching, data conversion, indexing, single instancing, de-duplication,legal discovery, data classification, implementing storage policies,combinations of the same or the like.

As discussed, disclosed embodiments of the invention allow forreporting, diagnostic, maintenance and/or other tools to be applied tothe extracted data/metadata from heterogeneous data sources. Suchembodiments, thereby, advantageously allow for such tools to beperformed with fewer queries, since it is not necessary to have multipleavailable queries that each apply to only a single data format.

FIG. 4A illustrates an exemplary simplified menu 400 for displaying in acommon window a variety of source devices in an information managementsystem. In certain embodiments, the menu 400 is accessible through theuser interface 108 of the information management system 100 of FIG. 1.In particular, the menu 400 comprises a “drill down” menu in a treestructure format that identifies various client devices 104 associatedwith the information management console 102. In certain embodiments, thehierarchical structure of the menu 400 can advantageously reflect thephysical or logical configuration of the information management system100. For instance, in certain embodiments, the user is able to select anicon or node of the tree structure that is associated with a clientdevice to determine if additional sub-clients or other computingdevices, such as media devices, are further associated with thehigher-level device.

For instance, as illustrated in FIG. 4A, one of the client devices 104of the information management system 100 is identified in the menu 400as “GALAXY 1.” This client device is further associated with threebackup tapes (i.e., “TAPE 1,” “TAPE 2” and “TAPE 3”). In certain furtherembodiments, when the user selects a particular node 430 (such as an endnode), the user is provided with a number of options or data utilitiesthat are available for data associated with the node. For instance, theuser may be provided (e.g., through a pop-up window, a drop-down list,or the like) with operation menus to assimilate jobs into theinformation management console 102; archive, search, or index data ofthe node; combinations of the same or the like.

As can be seen from the menu 400, the user interface can include bothbackup and non-backup client devices in a single view. Moreover,although the menu 400 provides one example of a graphical interface fora user, other embodiments of the invention can use different types ofuser interfaces including, but not limited to, windows, models, lists,tables, links, browsers, combinations of the same or the like.

For example, FIGS. 4B-4F illustrate exemplary screen displays usablewith the user interface 108 of the information management system 100 ofFIG. 1. In particular, FIG. 4B shows an exemplary screen display 440 forreceiving and displaying information to a user regarding the informationmanagement system 100. For instance, a side browser window 442 lists aplurality of client devices 104 that are associated with the informationmanagement system 100, namely “rajgad,” “vexsk3,” and “w2k7x64.”

Each of these client devices 104 further comprises data that ismanageable by the information management console 102 through one or moredata connectors 106. For instance, client device “vex2k3” furthercomprises multiple types of data stored thereon, including at leastEXCHANGE database, EXCHANGE mailbox and file system data. The browserwindow 442 further illustrates that a data connector 106 (i.e.,“vex2k3_netapp_snap_exchange_1”) for obtaining information regarding aNETAPP snapshot of EXCHANGE data on the “vex2k3” client device.

Lower browser window 444 further displays information regarding one ormore jobs being executed with respect to data of the client devices 104.For instance, the window 444 illustrates details regarding a backupoperation being managed by the information management console 102 withrespect to EXCHANGE data stored on the “w2k7x64” client device.

FIG. 4C shows an exemplary screen display 450 further illustrating adialog box 452 for selecting a particular data connector 104 for data ofa client device 104. FIG. 4D shows an exemplary screen display 460further illustrating a dialog box 462 for configuring properties withrespect to a selected data connector 106. For instance, the dialog box462 can receive configuration information usable to extract, access,process, mine or otherwise manipulate data on the client devices 104.

FIG. 4E illustrates an exemplary screen display 470 including a dialogbox 472 for managing and/or scheduling one or more jobs with respect todata on the client device 104. For example, the dialog box 472 can beused to perform one or more storage operations on data of one or more ofthe heterogeneous client devices 104, including, for example, backupoperations, snapshot operations, archive operations, informationlifecycle operations, combinations of the same or the like.

FIG. 4F shows an exemplary screen display 480 further illustrating adialog box 482 for configuring properties with respect to managing dataon a particular client device (i.e., the EXCHANGE mailbox on the“vex2k3” client device). For instance, the dialog box can be used toobtain information and/or instructions for mining data on the clientdevice(s) 104.

FIG. 5 illustrates a flowchart of a process 500 for managing data in aninformation management system having data stored on a plurality ofheterogeneous data sources. In summary, the process 500 provides a viewof exemplary steps involved in ingesting data from heterogeneous datasources, such as in a variety of backup formats, preparing the data, andmanaging the data, such as through processing, mining, copying or thelike. For exemplary purposes, the process 500 will be described withreference to the components of the information management system 100 ofFIG. 1 and the information management console 302 of FIG. 3.

The process 500 begins with the data ingestion procedure, illustrated byBlocks 505 to 520, during which data and/or metadata is extracted from aplurality of heterogeneous client devices 104 in the informationmanagement system 100. At Block 505, configuration information isreceived by the information management console 102 for one or moreclient devices installed, or being installed, in the informationmanagement system 100. In certain embodiments, the configurationinformation is received from a user or system administrator through theuser interface 108. For example, the configuration information mayinclude one or more of the following pieces of information regardingeach client device to be installed: instance name, security information,authentication information (e.g., username and/or password),combinations of the same or the like. In yet other embodiments, at leasta portion of the configuration information can be automatically obtainedfrom the client device 104 upon installation.

Based on the obtained configuration information, one or more dataconnectors or agents 106 are installed on the client device(s) 104(Block 510). As discussed above, in certain embodiments, each connector106 is customized to handle a particular data format and/or device andis push-installed to the specific client device 104 based on theconfiguration information. For instance, different data connectors 106can be associated with SYMANTEC NETBACKUP, MICROSOFT EXCHANGE, and thelike.

At Block 515, the data connector(s) 106 extract actual data and/ormetadata from the respective client device(s) 104. For simplificationpurposes, actual data and metadata will be referred to herein withrespect to this process 500 as “data.” For example, the dataconnector(s) 106 can read the client device data directly, scrub logfiles, and/or communicate with one or more APIs and/or nativeapplication processes to extract the data.

It will also be appreciated that, although the data extraction processis illustrated in a single block, the data extraction can proceed on acontinuous, periodic, and/or requested basis as data on the clientdevice 104 is modified. Thus, Block 515 may continue concurrently withsubsequent blocks of the process 500. For instance, upon the initialscan of the client device 104 following installation, the dataextraction process may be relatively resource- and time-intensive, butfollowing the initial scan, subsequent data extraction may be lighter ornon-existent (e.g., for decommissioned client devices 104).

At Block 520, the data connector 106 stores the extracted data in thenormalized data structure(s) 109, such as an XML file managed by theinformation management console 102.

Subsequent Blocks 525 to 545 illustrate the preparation and managementof the extracted data. At Block 525, the process 500 generates andstores value-added data in the normalized data structure 109. Forinstance, the data connector 104 and/or information management console102 can supplement additional metadata to the data structure 109regarding the extracted data, wherein the additional metadata isnon-native to the application that originally generated the extracteddata.

At Block 530, the process 500 presents a unified view through the userinterface 108 of the data stored on the various, heterogeneous clientdevices 104. In certain embodiments, the unified view is based on thenormalized information stored in the data structure 109, which includesboth extracted data and value-added data. In certain embodiments, thenormalized data in the data structure 109 advantageously provides ameans for executing a query with respect to data of multiple,heterogeneous client devices, rather than querying each deviceindependently.

At Block 535, the information management console 102 receives inputregarding a desired data management utility or processing operation. Forinstance, the input can be indicative of one or more of the following:data mining, archiving, stubbing, data migration, content indexing,document level backup mining (e.g., for non-backup servers such asSHAREPOINT, SQL, EXCHANGE servers), snapshots, restoring data,searching, combinations of the same or the like. In certain embodiments,the requested operation is based at least in part on one or more rulesor policies, such as stored in the management database 110.

Based on the request and the data involved, the information managementconsole 102 prepares the data for the request operation (Block 540). Forinstance, in certain embodiments, the data request may concern datastored in a snapshot. In such embodiments, the information managementconsole 102 can mount the snapshot and copy the data to a storage medium(e.g., tape), a staging area, the secondary storage 111 or the like.

In yet other embodiments where the subject data is stored on a tape orother media, the data preparation can utilize a proxy server to accessthe data and transfer a copy of the data to a staging area. In furtherembodiments, the data preparation can comprise establishing a path to aparticular library, folder or other data repository that stores thesubject data.

After the data is prepared, if needed, the process 500 performs therequested data operation (Block 545). In the case of restoring databacked up on one of the client devices 104, the process 500 can utilizethe native backup application of the client device 104, thecorresponding data connector 106, the information management console102, a customized data agent or the like, to restore the data.

FIG. 6 illustrates a flowchart of an exemplary process 600 forprocessing data requests in an information management system. Forinstance, the process 600 expands on Blocks 530 to 545 of the managementprocess 500 of FIG. 5. In summary, the process 600 allows a user toperform one or more operations on data stored on one or more clientdevices through a common information management console. For exemplarypurposes, the process 600 will be described with reference to thecomponents of the information management system 100 of FIG. 1 and theinformation management console 302 of FIG. 3.

The process 600 begins at Block 605, wherein the user interface 108displays a unified view of a plurality of heterogeneous client devices104. In certain embodiments, the client devices 104 comprise at leastmultiple devices that are associated with different backup applicationsor vendors. For instance, the client devices 104 can comprise serversthat back up data to a plurality of storage media at the same ordifferent locations. In yet further embodiments, the client devices 104can comprise database servers and/or non-backup servers, such as anEXCHANGE server, a SHAREPOINT server, SQL SERVER, combinations of thesame or the like.

In certain embodiments, the user interface 108 advantageously displaysgraphical representations, such as icons, of the client devices 104(e.g., icons) and/or available data management operations in a unifiedview, such as via a single window (see, for example, menu 400 of FIG.4A). In further embodiments, the user interface 108 can displayinformation regarding the client devices 104 and/or the data storedthereon, such as for example, media type, storage format, storage costs(actual and/or predicted), storage allocation (actual and/or predicted),combinations of the same or the like.

At Block 610, the user interface 108 receives a selection of at leastone of the client devices 104. For instance, such a selection may bemade by a user clicking on an icon representing the particular clientdevice 104 in the information management system 100. At Block 615, theuser interface 108 displays the files, folders, databases, or other dataelements that are stored on the selected client device 104. In certainembodiments, the display is in the form of a list, a tree structure orthe like.

At Block 620, the process 600 receives a request to perform a dataoperation on one or more data elements of the selected client device(s)104. In certain embodiments, the data request is received from a uservia the user interface 108. In yet other embodiments, the data requestis automatically generated based on one or more storage policies,schedules, user preferences or the like. In certain embodiments, therequest may concern a request to restore data that has been backed up byone or more client devices 104. In other embodiments, the request cancomprise performing one or more of the following data operations:archive, backup, copy, search, classify, browse or the like.

It should be appreciated that, although Blocks 605, 610, 615 and 620 areshown and described with reference to separate acts, one or more of theaforementioned blocks can be combined into a single block or act. Forinstance, in certain embodiments of the process 600, a requested dataoperation can identify a client device 104 and/or one or more dataelements on the client device 104 that are the subject of the dataoperation without requiring the process 600 to receive a separateselection of the client device 104 (Block 610) or to display the filesand/or folders of the client device 104 (Block 615).

At Block 615, the information management console 302 determines if thedata operation request concerns data that is on a non-backup media(e.g., a database). For instance, in certain embodiments, theinformation management console 302 can access information stored in themanagement database 110 to determine the format(s) of the subject data.In yet other embodiments, the data connectors 106 can be configured todetermine what type of data in on the client devices 104.

If the data request concerns a backup medium, the information managementconsole 302 accesses the device(s) storing the data (Block 630). Forinstance, the information management console 302 can mount, if needed, aparticular media device and read the backed up data stored thereon. Inyet other embodiments, the information management console 302 can accessthe data on the device through a proxy server, a network path or othermeans.

At Block 635, the information management console 302 performs therequested data operation. For instance, in the case of a restoreoperation, the restore module 328 can access backed up or archived datato restore the data to a workable format. In yet other embodiments, therestore module 328 can utilize a data connector 106 and/or a nativeapplication of the client device to restore the data.

At Block 640, the results of the request operation can be optionallystored in a memory, such as in the management database 110, for futureuse by the user and/or the information management console 302. Forexample, the cataloged information can be subsequently used to locateand archive certain data from the media devices according to one or morestorage policies (e.g., retention policies).

If at Block 625 the information management console 302 determines thatthe data of interest is on non-backup media, the process 600 moves toBlock 645. For instance, such data can comprise databases related toapplications such as EXCHANGE, SHAREPOINT, ACTIVE DIRECTORY or the like.In such situations, the mining module 330 of the information managementconsole 302 copies the data to a staging area to unravel the databasedown to its individual elements (e.g., mailboxes, email messages,storage accounts, user accounts, etc.) (Block 650).

In certain embodiments, Blocks 645 and/or 650 comprise decrypting and/ordecompressing the non-backup data. In yet further embodiments, theprocess 600 can comprise using a proxy to restore the data. For example,copying and mining the data can comprise emulating the nativeapplication (e.g., EXCHANGE, ORACLE) relating to the data of theparticular files/folders being copied to the staging area. In certainembodiments, such emulation includes presenting a user interface and/orone or more APIs to access and/or manipulate the data as if the nativeapplication was installed on the proxy machine(s). However, instead ofmaintaining the data in the native format, once it is processed, thedata can advantageously be converted into a file format that is moreeasily managed.

The processes described above can be especially advantageous inembodiments of the invention wherein vendors periodically change theirfile formats and/or the file format is decommissioned. For instance, byemulating the native application on a proxy machine, the system is notrequired to maintain copies of new versions of the native applicationeach time the application changes.

Following Block 630, the process 600 continues with Block 635 to performthe requested data operation.

FIG. 7 illustrates an exemplary block diagram of an informationmanagement system 700 for combining disparate data of multiple clientdevices, according to certain embodiments of the invention. Forinstance, the information management system 700 can be configured toaccess data in different formats on different client devices and run ormore common jobs on the data. For exemplary purposes, the system 700will be described with respect to performing migration operations on theclient data in different backup formats. However, it will be understoodthat the system 700 can be configured to perform other types of datamanagement operations, including, but not limited to, backup, archive,snapshot and like operations.

As shown, an information management console 702 communicates with aclient A 704 a via a data connector 706 a and a client B 704 b via adata connector 706 b. In certain embodiments, client device 704 acomprises data in a first backup format, and client device 704 bcomprises data in a second backup format. Moreover the first and secondbackup formats can comprise different properties such that a nativeapplication operating in accordance with the first backup format is notcapable of working with the second backup format. For example, the firstbackup format may comprise a NETBACKUP backup format, while the secondbackup format may comprise a NETWORKER backup format.

According to the methods previously discussed herein, the informationmanagement console 702 can receive metadata from each of the clientdevices 704 a, 704 b via the connectors 706 a, 706 b for storage in oneor more normalized data structures 709, examples of which areillustrated in FIGS. 2A-2D. For instance, the console 702 can populatevarious fields of the data structure(s) 709 with metadata received fromthe connectors 706 a, 706 b. For instance, such fields may dictate thelast access and/or modified time of the client data.

Moreover, the console 702 can further populate the data structure(s) 709with additional, value-added data generated by the console 702. Forinstance, information regarding the cost of a particular medium occupiedby the data and/or the overall storage capacity of the medium can beadded to the data structure 709. In such embodiments, the data structure709 storing the metadata for the data in first backup format (i.e., onthe client device 704 a) has the same fields and/or structure as thedata structure 709 storing the metadata for the data in second backupformat (i.e., on the client device 704 b).

The information management console 702 can further maintain and/oraccess one or more storage policies, jobs or other scheduled operationsfor manipulating and/or extracting data from both the client devices 704a, 704 b. In certain embodiments, the console 702 is configured,according to one or more storage retention policies, to migrate backupdata that has not been accessed for a particular period of time. In suchembodiments, when the threshold time has passed, the informationmanagement console 702 can instruct the connectors 706 a, 706 b toobtain and send the backup data to the console 702. In certainembodiments, each of the connectors 706 a, 706 b can access the data inthe first and second backup formats, respectively, via one or morepublished APIs.

Based on the metadata available from the data structure 709, theinformation management console 702 can store the backup data acquiredfrom the two client devices 704 a, 704 b to the same secondary storage,such as a tape drive. Such is especially advantageous in systemsattempting to reduce the costs of primary storage (e.g., on the clientdevice(s) 104)). For example, a storage retention policy can beestablished to migrate data from more expensive storage media (e.g., aNetApp filer or NAS) device to a less expensive storage media (e.g.,tape) after a predetermined period of time.

In certain further embodiments, the information management console 702can further provide one or more reports via a user interface 708regarding costs of storage within the information management system 700.For instance, the user interface 708 can be configured to display a piechart or other visual representation of a cost per unit of storage(e.g., dollars per gigabyte) for the various client devices 704 a, 704 bwithin the system 700. Based on these reports, the user can be informedhow much storage is costing and/or predicted to cost and be providedwith the option of establishing storage criteria for managing themigration of data within the system 700. Various other reports can beprovided.

Systems and modules described herein may comprise software, firmware,hardware, or any combination(s) of software, firmware, or hardwaresuitable for the purposes described herein. Software and other modulesmay reside on servers, workstations, personal computers, computerizedtablets, PDAs, and other devices suitable for the purposes describedherein. Software and other modules may be accessible via local memory,via a network, via a browser, or via other means suitable for thepurposes described herein. Data structures described herein may comprisecomputer files, variables, programming arrays, programming structures,or any electronic information storage schemes or methods, or anycombinations thereof, suitable for the purposes described herein.

Embodiments of the invention are also described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products. It will be understood that eachblock of the flowchart illustrations and/or block diagrams, andcombinations of blocks in the flowchart illustrations and/or blockdiagrams, may be implemented by computer program instructions. Thesecomputer program instructions may be provided to a processor of ageneral purpose computer, special purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions, which execute via the processor of the computer orother programmable data processing apparatus, create means forimplementing the acts specified in the flowchart and/or block diagramblock or blocks.

These computer program instructions may also be stored in acomputer-readable memory that can direct a computer or otherprogrammable data processing apparatus to operate in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including instruction meanswhich implement the acts specified in the flowchart and/or block diagramblock or blocks. The computer program instructions may also be loadedonto a computer or other programmable data processing apparatus to causea series of operations to be performed on the computer or otherprogrammable apparatus to produce a computer implemented process suchthat the instructions, which execute on the computer or otherprogrammable apparatus, provide steps for implementing the actsspecified in the flowchart and/or block diagram block or blocks.

While certain embodiments of the inventions have been described, theseembodiments have been presented by way of example only, and are notintended to limit the scope of the disclosure. Indeed, the novel methodsand systems described herein may be embodied in a variety of otherforms; furthermore, various omissions, substitutions and changes in theform of the methods and systems described herein may be made withoutdeparting from the spirit of the disclosure. The accompanying claims andtheir equivalents are intended to cover such forms or modifications aswould fall within the scope and spirit of the disclosure.

What is claimed is:
 1. A method for processing data from a plurality ofclient devices, the method comprising: providing a graphical userinterface including on-screen graphics representing a plurality ofclient devices in a storage network system, wherein the plurality ofclient devices comprises one or more first client devices storingsecondary copy data in different secondary copy formats and one or moresecond client devices storing primary copy data generated by one or moreapplications executing on the second client devices, wherein thegraphical user interface is provided via a management console executingon a computing device in network communication with the client devices;receiving a data operation request including (i) a first selection of atleast one second client device of the second client devices, and (ii) afirst data management operation, the first data of the at least onesecond client device generated by a first application of the one or moreapplications; using a data connector that is in communication with themanagement console, obtaining the first data of the at least one secondclient device from storage associated with the at least one secondclient device, the data connector corresponding to a file formatassociated with the first native application; emulating, using one ormore processors and on the proxy device, the first application; and withthe emulated application, performing the first data management operationon the obtained first data.
 2. The method of claim 1, wherein the firstapplication comprises a database application.
 3. The method of claim 1,wherein the first application comprises an electronic messagingapplication.
 4. The method of claim 3, wherein the first data managementoperation comprises identifying a first mailbox within the first data ofthe at least one second client device.
 5. The method of claim 1, furthercomprising, based on said processing, updating a catalog comprisingmetadata corresponding to the first data of the at least one secondclient device.
 6. The method of claim 1, additionally comprising:receiving a second data operation request including (i) a secondselection of at least one first client device of the first clientdevices, and (ii) a second data management operation; and performing thesecond data management operation on the backup data of the at least onefirst client device.
 7. The method of claim 6, wherein the first andsecond data management operations comprise storing, respectively, thefirst data of the at least one second client device and the backup dataof the at least one first client device in a common data format.
 8. Themethod of claim 7, wherein the common data format comprises an archiveformat.
 9. The method of claim 6, wherein at least one of the first andsecond data management operations is performed based on a migrationstorage policy.
 10. The method of claim 9, wherein the migration storagepolicy evaluates a cost of storing the data on the at least one firstclient device and/or the at least one second client device.
 11. A systemfor administering a storage network comprising heterogeneous clientdevices, the system comprising: a plurality of first client devicesconfigured to store backup data in different file formats; a pluralityof second client devices configured to store non-backup data indifferent file formats; a proxy device comprising at least one emulatednative application executing thereon; a secondary storage device; amanagement console executing on a computing device in networkcommunication with the first and second client devices, the proxy deviceand the secondary storage device, wherein the management console isconfigured provide a single interface including on-screen graphicsrepresenting the first and second client devices, and wherein themanagement console is configured to, receive a first data operationrequest with respect to the non-backup data stored on at least one ofthe second client devices, and copy the non-backup data of the at leastone second client device to a staging location associated with the proxydevice, and wherein the proxy device is configured to process thenon-backup data of the at least one second client device with the atleast one emulated native application and copy the processed non-backupdata to the secondary storage device.
 12. The system of claim 11,wherein the management console is further configured to: receive asecond data operation request with respect to the backup data stored onat least one of the first client devices; process the backup data of theat least one first client device; and copy the processed backup data tothe secondary storage device.
 13. The system of claim 12, wherein thenon-backup data of the at least one second client device and the backupdata of the at least one first client device are stored on the secondarystorage device in a common file.
 14. The system of claim 12, wherein atleast one of the first and second data operation requests is based on astorage policy to migrate data to a less expensive storage medium aftera predetermined period of time.
 15. The system of claim 11, furthercomprising: a plurality of first data connectors in communication withthe management console and corresponding to the different file formatsof the backup data; and a plurality of second data connectors incommunication with the management console and corresponding to thedifferent file formats of the non-backup data, wherein the first andsecond data connectors are configured to obtain data from, respectively,the first and second client devices.
 16. The system of claim 15, whereinat least one of the first data connectors is configured to obtain backupdata in a first file format from multiple ones of the first clientdevices.
 17. The system of claim 15, wherein multiple ones of the seconddata connectors are configured to access non-backup data on a single oneof the second client devices.
 18. The system of claim 11, furthercomprising a migration storage policy, and wherein the data operationrequest is based at least in part on the migration storage policy.